Strong Scaling Study

Adam Lyon, Muon g-2 IRMA Analysis, Fermilab, October 2020

This notebook examines strong scaling properties of Julia IRMA jobs making just one plot. I ran jobs with 2,3,4,5,6,7,8,9,10,12,15 and 20 nodes and always with 32 tasks per node. If I examine the timings of a part of the Julia run that includes opening and reading the HDF5 input file, creating the histogram, and running MPI.Reduce and MPI.Gather, I see expected strong scaling. The jobs get faster with more nodes as each task has less of the file to read. If I examine the total elapsed time reported by the batch system, then the scaling is less clear. It almost appears that more nodes make Julia run more slowly. Speculation suggests that perhaps there is contention for loading packages. Next steps could be to try PackageCompiler.jl to make a Julia "app" with fast startup time.

This notebook answers issue Analyze results from Strong Scaling jobs #18 and code may be found in PR #20. This file is IRMA/analyses/018_StrongScaling/StrongScaling.jl.

What is this notebook?

This is a Pluto.jl notebook and the code here is written in Julia. This is like a Jupyter notebook, but with important differences. The most important difference is that the results appear above the code. Another important difference is that Pluto.jl notebooks are reactive. This means that unlike Jupyter notebooks, Pluto.jl notebooks are always in a consistent state. The notebook keeps track of the cell-to-cell dependencies and when a cell changes, the dependent cells update at the same time. This means that while you are looking at a static html representation of the notebook, you can be assured that the notebook is consistent and up-to-date. You'll see that some results have a little triangle next to them. Clicking on that will open an expanded view of the results.

The organization of this notebook is that the main results are replicated at the top, with discussion, in the Results section. The plots are stored in variables which you can see below the plot. You can look in the Code section, which has all code for this notebook, to see how the plot was made.

Introduction

This notebook examines strong scaling properties of my Julia IRMA jobs that make one plot.

On 10/22, I ran IRMA/jobs/003_StrongScaling/strongScalingJob.jl from commit 82b715b answering issue #3. This job reads in the cluster energy data from Muon g-2 era 2D and makes a plot of that energy for all clusters. The data is split evenly among all the MPI ranks. I've tried 2 nodes through 10 nodes. I always choose 32 tasks per node (advice from Marc Paterno). I ran the jobs in the debug queue.

On 10/27 I ran three more jobs with 12, 15 and 20 nodes (still 32 tasks per node) respectively. These jobs ran in the debug queue.

Note that on 10/25, I ran jobs with 12, 15 and 20 nodes in the regular queue. Note that I ran the 12 node job twice, the first time it ran I got a strange error, I think due to the CSCRATCH filesystem crashing (It's been a bad month for Cori). The second time I tried it, it ran fine. I ran these jobs in the regular queue, because the debug queue was very full. Because the elapsed time and memory usage looked strange, I replaced these runs with runs in the debug queue (see 10/27 above). See a comparison in Code of these runs in debug and regular queues.

All jobs ran on Haswell. Data came from CSCRATCH.

I recorded MPI timings from the Julia run. I also dumped SLURM accounting information for analysis.

Results

There are several types of results.

Histogram comparison

Each rank makes a histogram of cluster energy. All of these histograms are sent to the root rank with MPI.Gather and saved in the output file. Furthermore, I "reduce" the histogram by merging them into one with MPI.Reduce. See Histogram Comparison in the Code section where I compare these histograms to be sure that the reduced one is correct. Note that I'm using a static histogram defined in the IRMA.jl package that MPI can manipulate directory without the need for serialization/deserialization. See the comparison in Code (see above). The tests worked fine.

MPI Timing information

I record the time in the job with an IRMA Stopwatch. The stopwatch uses MPI.Wtime under the hood. The times are recorded as follows.

Label	Meaning
start	After packages are loaded, functions are defined, and MPI.Init call
openedFile	After the h5open statement
openedDataSet	After the energy dataset is opened (but no data read yet)
determineRanges	After the ranges to examine are determined with partitionDS
readDataSet	After the dataset is read (this reads the actual data for the rank)
makeHistogram	After the data is histogrammed
gatheredHistograms	After all the histograms have been gathered to the root rank
reducedHistograms	After the histograms have been reduced to the root rank
gatherRankLogs	After the rank's log info has been gathered to the root rank

Timing of anything before start is not recorded.

Let's look at the timing plots. Note that you can see all the timing plots in the Code section. Some representative plots will be reproduced here.

Here is the run using two nodes (and 32 ranks per node).

First, you see that each rank read 263,062,959 rows, or one less. Just opening the file took over 5 seconds, with the root rank taking a little bit longer. Reading the dataset itself takes the majority of the time. The spread of times over ranks is rather interesting. Some ranks took a bit longer to make the histograms. Some ranks were signficantly slower in the MPI.Gather, though the structure perhaps shows some clever consolidating in MPI. Reducing the histograms involves every other rank taking about a second longer than the others. Gathering the rank logs is very fast. You see all of this structure in the total MPI time.

Let's look at ten nodes

There's quite a bit of structure in these plots. Note that for readDataSet, where the data is actually read in, some nodes seem appear to be faster than others. Given that the reading is fast, but its time still dominates, the structure of the reducedHistograms becomes clear in the total time.

Here are twenty nodes